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ABSTRACT 

To identify and describe secondary vocational 
education programs that have been successful in increasing the 
placement rate when compared with other courses of 
programs were selected through a literature search f 
telephone inquiries, personal contacts, and other 
evaluation of each program was studied, particularly 
graduates, in addition, on*sife evaluations were made 
staff of evaluation problems in 30 programs. The project staff 
unable to show that any of the program met the study 1 * 
success. Nearly one-half of the programs did 
followup information on their graduates, and no trends 
detected nor suitable comparisons made of those 
programs that had nearly complete followup records. In order 
obtain detailed Information on graduates, more complex, fine-i 
followup would be required. It was also 
group be selected for 
to select a compa 
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SUMMARY 



The aims of this study (which was a sequel to two similar studies 
of compensatory education programs) were to identify, select, analyze, 
and describe vocational education programs at the secondary level that 
had been successful in increasing the total placement rate when compared 
with other courses of instruction. The. total placement rate was defined 
as the combined rate of placement: of graduates in employment and further 
education. 

It was expected that the study would provide descriptions of success- 
ful programs to serve as models for replication. 

Through a literature search, mail and telephone inquiries, personal 
contacts, and other means, a pool was established of 445 programs thought 
to be likely candidates for the study. Most of the programs in the pool 
were recommended by professional vocational educators, either at State 
or local levels, or in universities, who suggested that these programs 
were worth reviewing. The evaluation of each of the programs in the 
pool was studied. In particular, questions were asked about the follow-up 
of graduates. Some 30 programs were site-visited by AIR to examine their 
problems of evaluation at first hand. 

The AIR team was unable to show that any of the 445 programs met the 
study's criteria for success. Nearly one-half of these programs did not 
have comprehensive follow-up information on their graduates, making it 
impossible to determine the placement rates. An additional one-third of 
the programs had nearly complete follow-up records on their graduates, 
but no trends could be detected from their data, nor were suitable com- 
parison groups of non vocational graduates available for any of them. It 
became apparent that there were problems of evaluation design and data 
collection to be solved before any of the programs could show increases 
in total placement rate when compared with other courses of instruction. 

In fact, the criteria of increased total placement rates of graduates 
in employment and further education, stated in strictly quantitative terms 
were not at all useful in measuring the success of vocational programs. 
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Even if comparable, contemporaneous groups had existed, and even if com- 
plete follow-up of both groups of graduates had been possible, the cri- 
teria still would have to be supplemented by criteria which took into 
account qualitative placement factors, such as initial job earnings, on- 
the-job competency, job retention, progress within the job, and mobility 
within a range of jobs related to the training provided by the vocational 
program. 

It was concluded that to obtain this relatively detailed information 
on graduates, more complex, fine-grained follow-up would be required. 
Furthermore, to insure an adequate response to follow-up requests, it was 
recommended that a properly randomized group of manageable proportions be 
selected from the program for follow-up, and that the same procedure be 
applied to select a comparable, contemporaneous group of nonvocational 
graduates. This type of design would permit factors outside the program 
which may be unstable over time (fluctuations in the local labor market, 
for one) to operate on both groups. 

More comprehensive and fine-grained follow-up would permit far more 
detailed conclusions to be drawn about a program’s relative success than 
is possible based on the comparatively coarse, and subjective measures 
that are currently being applied. 

Since many local vocational program evaluators are apparently eager 
to adopt improved evaluation procedures such as those recommended in this 
report, it was further recommended that training in evaluation design be 
provided to local evaluators. 




















INTRODUCTION 



Background 

The aims of this study were to identify and describe successful 
vocational education programs in American, secondary schools. The programs 
described were to serve as models for school systems seeking to improve 
the quality of the education they offer. 

These aims were established following the pattern of earlier studies 
by A III, during which selected successful compensatory education programs 
were described (see Uawkridge , Clial up sky , and Roberts, 1968; Hawkridge, 
Campeau, DeWitt, and Trickett, 1969). The criteria of success for the 
compensatory programs were different from the criteria for this study, 
being based on cognitive achievement as measured by standardized tests. 

No compensatory program was accepted for description unless data available 
indicated that pupils in the program had achieved statistically signifi- 
cantly better gains on standardized tests than had controls, or than 
national normative figures. 



The aims of the present study were also influenced by a study by 
Decker (1968). Decker's study aimed at identifying and analyzing effec- 
tive vocational education programs for disadvantaged students in secondary 
schools, but used as its criteria of effectiveness reductions in dropout 
rates and increases in standardized test scores. In Decker's work, over 
200 programs were examined but none could be identified as "truly 
exemplary." For the present study, the scope was widened to include 
programs not specifically for the disadvantaged, and the criteria were 
altered, as will be. shown under the next: section of this report. 

Limits of the Studv 

, ■ , — -r t r - - - 

Vocational education programs have been funded by several Federal 
agencies, as well as by States, local school districts, and private bodies 
such as foundations. The study did not include those programs funded by 
the Department of Labor, the Office of Economic Opportunity, or similar 
labor-oriented agencies at State or local levels, but restricted itself 
















to programs operating in publicly or privately owned schools. Thus pro- 
grams such as the Neighborhood Youth Corps and those funded under the 
Manpower Development and Training Act were omitted. 

In order to avoid replication of effort, the study did not collect 
data from programs already subject to scrutiny by Project METRO, a major 
survey now being conducted, and referred to below in the discussion on 
normative data. 

The study was further limited to vocational education programs in 
secondary schools, and specifically to programs serving pupils in grades 
nine through twelve. (In an ungraded program, the age of 18 years was 
taken as the upper limit.) Programs that terminated before 1964 were 
not considered. 

Programs mainly developed for social rehabilitation of handicapped 
or delinquent youth were omitted from the study as it seemed very likely 
that any described would not provide models suitable for generalization 
to pupils not in need of such rehabilitation. 

No geographical limits (within the nation) were placed on the study 
by restraints of time or funds. If the initial screening had yielded 
more programs which looked likely to meet the criteria, they could have 
been visited and closely examined. 

Naturally, the study was limited most by its criteria. The primary 
criteria were related to follow-up of graduates of vocational programs; 
thus the search for programs which might meet these criteria was directed 
towards the more likely candidates, as will be demonstrated under Methods 
and Procedures below. 
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Re Xate d Research 

Tills section of the report will review literature dealing with eval- 
uation of vocational education. Its main purposes are to point out the 
present state of evaluation, in. vocational education and the kinds of 
problems which have frustrated such assessments. 

A few readers may already be familiar with this literature and with 
the problems which characterize vocational program evaluation, but others 
are urged to read this review. It contains some very important points 
which are related to the way the AIR study was planned and the way it 
turned out. 

The review deals first with general statements about evaluation in 
vocational education and with evaluation criteria and procedures which 
have been proposed in the literature. 

Next, the review describes specific evaluation methods used in three 
large-scale studies of vocational education. Findings are not dealt with 
at all because it was felt that for the purposes of this report, back- 
ground in methods and procedures which have been used in evaluating voca- 
tional education would be more relevant and useful. Accordingly, descrip- 
tions of specific evaluation techniques are deliberately detailed to help 
the reader appreciate the care with which these three major studies were 
planned and the difficulties in implementation which were bound to affect 
the credibility of their findings. 

Reviews of evaluation reports of single programs have not been 
included here, but three case studies of programs visited during this AIR 
study can be found in Appendix A. 

Finally, this literature review is not a critical one. No attempt 
has been made to judge the success or failure of the evaluations or of 
the programs whose evaluations are described. 

G eneral statements on evaluation in vocational education . Coster 
and Ihnen (1968) reviewed program evaluation in vocational education and 
concluded that not only have "value judgments . . . weighed heavily in 
the assessments of objectives . . . [but] further, objectives have been 



v"- 

a® 












' v '■ V- : : ■ V iV- . : 

; ;.*> . . : ■ 5 w\: ■. ' ■■ !&;:?: ■ v :v? 









•W'-IMflpfH;**. 



V 






^ ■»^ ,, ^s wi '' (r ’ w ' *’° a< ’ ( * ^ * fc 5 , v« r- L *q 1 a7(f u_T^v> -**. 



sit a tod in rational rather than empirical terms . Rarely have objectives 
been stated as measurable products involving a component oi time [p. 418). n 
The evaluation oi vocational education, these authors charged , has been 
subjective rather than objective and has reflected more concern for the 
process of vocational education than for its product. 

In 1968, the. Advisory Council of Vocational Education published its 
general report (Advisory Council, 1968). Although this report covered 
a broader area than the Coster and Ilmen article, its conclusions with 
respect to program evaluation ware very similar. For example, the report 
suggested that "objective dimensions should he developed at the Federal 
level, involving appropriate State and local representatives in order to 
provide an objective evaluative system that can he used in each of the 
States and which can be combined into a national evaluation [p, 139]." 

This suggestion indicated that in fact little work on an "objective 
evaluative system" had yet been done. In fact, the Council stated that 
because of the reporting systems used in vocational education, "it is 
difficult to assess exactly what has taken place [p. 139]." 

The Council also recommended that each State conduct a periodic and 
statewide review and evaluation of its programs. "As nearly as we can 
determine, only a very few States have conducted a formal evaluation of 
their programs since 1917 [p. 203]." 

The. report of the Council showed that the. funds allocated to research 
in vocational education were used for a wide range of activities, only one 
of which was evaluation, and that the actual appropriations have fallen 
short of the 30% originally allocated by the Vocational Education Act. 

In 1968, for example, only 6.8% of the funds were appropriated. In 1967, 
the appropriation was only 4.8% of the funds. 

Sharp and Krasnegor (1966) surveyed the use of follow-up studies as 
evaluation tools. They found that follow-ups were common in some parts 
of the country but virtually nonexistent in others. Moreover, when follow- 
ups were conducted in certain regions, they often omitted some of the 
programs. Thus, follow-up coverage has been uneven. Graduates of certain 
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curricula such mu trades and industry or agriculture were moat likely to 
bo followed, whereas graduate*) of other programs (e.g, . , home economics, 
health occupations) were 3 oast likely to he followed , 

In addition, follow-up .studies often did not yield as much informa- 
tion as they could, Sharp and krasnegor noted that follow-ups would he 
more meaningful if vocational graduates were matched with similar non- 
vocational graduates. This might enable researchers to determine the 
effects of the different curricula , Moreover > most follow-up studies 
described the high school graduate at a certain point in time. Thus, 
no data about trends were available. If follow-up studies were conducted 
at regular intervals, researchers might have a hotter idea of the long- 
range effects of vocational education. The authors presented a model 
for effective follow-up studies which they hoped would produce more inter- 
pretable results . 



On a smaller scale, Kaufman and Lewis (3.968) surveyed reputedly 
successful vocational programs in three selected cities in Pennsylvania. 
A team of specialists visited the schools and prepared reports dealing 
with six aspects of the schools’ programs. Kaufman and Lewis reported 
that for each of the programs the weakest:, aspect was evaluation, taking 
evaluation to mean follow-up studies. According to the investigators, 
the evaluations conducted by these better- than -aver age programs were not 
done satisfactorily or in a systematic manner. 



Evaluative criteria and procedures proposed in th e literature . The 
need for evaluation procedures to be built into a program from its inception 
was emphasised by Sweeny (1966), who said that, programs should be planned 
so that they insure adequate feedback to the teacher about the effectiveness 
of the curricula. Sweany suggested various goals and characteristics of 
vocational education programs which might be evaluated. 

Bushnell (1966) also .stressed that only through evaluation can voca- 
tional education be improved. He pointed out that without evaluation one 
cannot select the most effective techniques and programs . Future areas 
of research on vocational education were suggested in his article. 

Loomis (1969) indicated that the long-range objectives of vocational 
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evaluation nynt < .*;* . lie specif hd eight factors which determined prop, nun 
effectiveness, including the* relevance of high school training to formal 
cm** the job training and the oxteut to which tin,* program relieved unemploy** 
mom, The initiation of follow-up procedures was suggested. 

Hotu specific suggestion.', have* boon a Ivon by other author**. Popham 
CH 960) reviewed the* ctiU rJn generally used by vocational -tucbnical 
teachers to evaluate curriculum material a, Ik* con aide-, tod the* criteria 
inadequate and proponed that new criteria emphasising behavior change 
and learning, be. developed and lined* 

Hurt and Barkley (1969) presented detailed suggestions concerning tlie 
assessment of various home economics programs* Their paper included a 
step-by-step analysis of the evaluation procedure. Types of evaluations 
and program criteria were also discussed* To determine the long-range 
effectiveness of home economics programs, the authors suggested using a 
longitudinal follow-up procedure. Graduates should bo evaluated at 
various intervals; they could then bo compared to a similar group of 
former students who did not take the program. Suggested reading for 
anyone interested in performing evaluations was also provided. 



Byram and McKinney (1968) have written a manual for evaluation of 
vocational education at. the local school level. They suggested many kinds 
of assessments, such as employer feedback and citizens’ committee appraisals. 
Follow-up as a form of evaluation -was considered in detail. Readers were 
told how to conduct a follow-up and what kinds of information to expect, a 
follow-up to provide. In addition, the. manual’s appendixes included forms 
and instruments developed as evaluation tools for local vocational schools. 



Methods e mpl oyed in major ev alu ation studie s. Kaufman, Schaefer, 
Lewis, Stevens, and Rouse (1967) surveyed nine communities in Maryland, 
Ohio, Pennsylvania, and New Jersey. The stated purpose of the study was 
to assess vocational education as compared with other high school offerings 
and to determine the extent to which vocational education was meeting the 
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needs of the students and the coainuinUy, The i nvoa Ligation. concentrated 
on three major :i navies i the adequacy of vocational education, the imago 
of vocational education, and vocatiioua.1 education for groups with special 
problems , Data on these issues were collected from four different sources. 
First of all, the programs themselves were examined by an independent 
team of exports. In addition, graduates of the vocational program were 
followed , interviewed, and compared to graduates of academic and general 
programs. Interviewers talked with employers who had hired graduates of 
the programs; union officials were also interviewed, 

The nine communities were chosen because they had vocational education 
programs commended by professional educators. (Kaufman et al. do not say 
whether the academic and general curricula used for comparison were also 
considered to be better than average.) Three large cities (500,000 or more 
population), three medium-sized cities (100,000-499,999), and three small 
cities (25,000-99,999) were selected. 

Selection procedures were different for high school graduates in the 
small and medium-sized cities than for high school graduates in the three 
large cities. In the small and medium-sized cities the investigators went 
through the school follow-up records of the graduates of classes 1960 through 
1964. All of the graduates who had gone on to college were eliminated. 

Of the remaining students, they selected 900 in each city to be sent letters 
requesting participation in the study. These students were selected so that 
approximately one-half of them would be vocational education graduates, 
one-fourth would be academic curriculum graduates, and one-fourth would be 
general curriculum graduates. From the 900 letters, approximately 600 
contacts were expected. This prediction proved to be quite accurate. 
Sixty-eight percent of those sent letters returned the enclosed reply card. 
The remaining 32% of the graduates were not sampled. In total, 2,831 
graduates were selected in this manner for interview (in person or by 
mail, see Chapter 2, page 5. 

In the three large cities the investigators anticipated that they 
would have difficulty locating and interviewing high school graduates. 

They therefore decided to combine the graduate and employer interviews by 
writing to various employers and asking to interview them and three of 





tlialr c i ]»vp3oyc‘(,\q. (The suleeU cm of employers i s <U scussed bct'law.) 
Employers who Imd agreed to the interview were miked t*o supply employees 
who had boon graduated from the selected high schools between 1960 and 
1964, If such graduates were not available, other graduates of the 
cities' schools wore selected, 

Kaufman et al, report that this technique yielded enough graduates 
in two out, of the three large cities but was less successful in the third. 
Therefore, in the third city a direct follow-up of high school graduates 
was conducted, This method was so successful that the number of inter- 
views obtained in this city was increased to 1,249 in order to offset 
the lower number in some of the other sample cities. Kaufman et al. 
do not report how this substitution was accomplished; however, since 
these investigators reported that they had adequately sampled the other 
two large cites (in which the average number of interviews was 591) , it 
seems likely that the increased sampling of the one large city offset 
the lower number of interviews in the six smaller cities. For the six 
smaller cities the number of interviews had averaged only 470. 



In the small and middle-sized cities, letters were sent to all 
major employers asking for interviews. Thirty-nine percent of the 
employers agreed to cooperate. 



In the three large cities, the investigators got lists of employers 
from the State employment security office. The employers were grouped by 
their major Standard Industrial Code classifications and the percentage of 
the labor force in each of the classifications was calculated. Employers 
were then selected from each classification in proportion to its contribu- 
tion to the total labor force. By this method, large businesses were 
selected for sampling* In order to sample smaller businesses as well, 
ten types of small businesses (e.g. , beauty shops, radio and television 
repair shops) were selected for sampling. Ten employers in each of these 
businesses were selected randomly from the telephone directory yellow 
pages and asked to cooperate. One hundred small businesses were contacted 
in this way. Since employers in large cities were asked both to be inter- 
viewed and to let three of their employees be interviewed, the researchers 
were not surprised when only 21% of the employers contacted in this way 
agreed to cooperate. • 
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IjU s C s of unions were not r 
investigators simply asked the 
furnish the names of any union 
cases the school officials did 
to obtain a list of thorn, the 
off i c i a .1 s i n th e. smal 1 c i 1.1 e s , 
15 to 20 in the large cities, 
in this attempt. In all, 90 
did not state which cities (or 
represented. The average rate 
f. r om c i ty t o c i ty . 



andoml ssed or controlled in any way. The 
school which was being evaluated to 
officials of whom they know. Xu some 
not know the unions and chid not; know where, 
investigators had. wanted to contact: 5 to 10 
10 to 15 in the medium-si sed cities, and 
They reported that they were unsuccessful 
:i.cia.1.s were contacted, Kaufman et; al. 
what si se cities) these union officials 
of reply was low, and this rate varied 



One of the two purposes of the survey was to study and assess voca- 
tional programs and the extent to which they were meeting the needs of 
the students and the community. In order to meet this objective, the 
investigators sent out an independent team of experts to evaluate the 
selected programs. The. team prepared a narrative report on each program 
studied. In addition, to provide for uniform assessment, an evaluation 
instrument was developed. In its final form it consisted of several 
hundred rating scales which applied to various aspects of the programs. 

The second objective of the study was to assess the vocational- 
technical curriculum as compared with other high school offerings. To 
make this assessment, high school graduates, employers, and union officials 
were questioned. Most of the high school graduates located were interviewed 
They were asked questions about their high school experience and about the 
jobs which they had held since graduation. The more mobile graduates 
were not interviewed; instead they were mailed a questionnaire which was 
essentially a simplified and shortened version of the interview. The 
questionnaire also included a set of questions on geographic mobility 
which was not included in the personal interview. More than 11,000 
graduates were mailed questionnaires and 30% of the questionnaires were 

m 

completed and returned. Kaufman et al. made a comparison of the mail and 
personal interview data to detect similarities in responses obtained by 
the two methods. 
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a tjriidtmlo \u\n in ter v Jawed , his direct. supervisor was asked to fill 
out. a Supervisor's Rating Scale. Thu supervisor ra tod the graduate on his 
job preparation and job performance, Graduates who were sent ques tionnaircs 
did not get a supej visor rating. 

To assess the. image of vocational education, employers and union 
officials wore interviewed whenever possible. In addition, questionnaire 
information was solicited from those who were interviewed. Labor loaders 
seemed reluctant to grant interviews. Kaufman et al. noted that even when 
interviews were conducted with union officials, many proved incomplete 
because the attitude scales which the interviewer left with the. union 
official were rarely completed and returned. Whereas 52% of these rating 
scales were returned by employers interviewed, less than 5% were returned 

by union officials. In total, 658 employers and 77 union officials were 
interviewed. 



Eninger (1965) selected and studied a random sample of 100 schools' 
geographically stratified throughout the United States. The study tried 
to fill two needs. First, Eninger' s survey covered all geographic regions 
of the country except Alaska and Hawaii in order to permit generalizations 
to be made about vocational education in the United States. Second, the 
survey began in 1963— the year that the Vocational Education Act was 
passed by Congress. Eninger studied graduates of the classes of 1953, 1958, 
and 1962. By describing vocational education as it existed before the 
1963 Act, Eninger sought t:o provide a norm against which changes in voca- 
tional education could be measured in the future. 

The study described the post-high school experience of vocational 
curriculum graduates. It compared vocational and academic course graduates 
from the same school and graduating class in terms of post-high school 
occupational, educational, and related experiences. In addition, the 
study considered several broad issues: How does choosing a general or a 

vocational curriculum affect a student's post-graduation experiences? 

What are the differences between graduates of vocational and comprehensive 
high schools? Does the size of the high school attended affect the 
Staduat© s future? In what ways do year of graduation, geographical region, 
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post-graduation mobility , and race affect post-high school occupational 
a n d ad u c a t i o an ,1 . ox p e r :i cm e e ? 

The survey covered 50 vocational and/or technical high schools and 
50 comprehensive high schools offering throe or more programs of trade and 
industrial education. Schools were stratified according to total enrollment, 
type of school (vocational-technical, vocational, technical, comprehensive), 
and geographic region. They were then randomly selected within each 
stratification category. 



Geographic stratification was done by portioning the United States 
into eight regions and calculating the percentage of the national population 
in each part of the country. Schools in each region were then sampled 
in proportion to their population. For example, New England contained 
10.8% of the national population and represented 11% of the students in 
Eninger's follow-up. The total enrollments of the sample schools and 
the type of school represented also closely paralleled the proportion of 
such schools in the total population. 

Questionnaires were mailed to 10,758 trade and industrial (T & I) 
curriculum graduates and 3,494 academic curriculum graduates. Only males 
were surveyed since they seemed more likely than females to be employed 
and seeking employment ten years after high school graduation. For each 
school the maximum number of graduates to be sampled was calculated. 



This number was a function of the number and kinds of vocational programs 
offered by each school. Whenever possible, the research personnel tried 
to sample an equal number of graduates from each T & I course offered. 



One survey questionnaire for academic graduates and another for 
vocational graduates were developed. A number of vocational educators 
were asked to review the questionnaires and to suggest necessary changes. 
The instruments were also pretested on local high school graduates to 
insure that the questions were easily understood and not open to misinter- 
pretation. The questionnaires lent themselves to a longitudinal or 
historical analysis rather than a point-in-time, description. For example, 



a graduate was asked to report on his job history rather than on his 
current job only. The questionnaires covered three general areas: occupa- 

tional history, educational history, and present interests, activities, 
and affiliations. 
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At this point, two pilot studies worn conducted to determine the 
best ways of getting a high return rate on. the questionnaires . Subjects 
for these pilot studies were .1933, 1938, and 1962 graduates of nearby 
comprehensive and vocational schools. In the first pilot study, they were 
sent either long or short questionnaires, and they were, either offered 
or not offered a material incentive to return the questionnaires. Neither 



the length of the questionnaire nor the offering of an incentive had a 
significant effect upon the return rate. However , the year of graduation 
did significantly affect the return rate of those graduates successfully 
contacted; the more recently a person had been graduated from high school, 
the more likely he was to return the questionnaire. 

' The second pilot study compared (a) the. effects of the high school 
letterhead versus the research institute’s letterhead in the introductory 
letter and (b) the effects of a closely massed mailing schedule of six 
reminders yersus a more widely spaced schedule. The research letterhead 
and the massed mailing schedule were found to be effective for increasing 
the return rate. 




The. information reported in the main study was based on a 50.5% 
return rate from vocational graduates and a 51.5% return rate from 
academic graduates. Those people who were sent questionnaires had been 
carefully selected to be representative of vocational and academic 
curriculum graduates. However, since half of the selected graduates did 
not reply, it was quite possible that those who did complete the question- 
naire were not totally representative of vocational and academic education 
graduates. Eninger considered this problem carefully and in detail. An 
analysis of the returns revealed several biasing factors. 

As the pilot tests had indicated, the more recent the student’s 
graduation, the more likely he was to be sampled. Graduates of the 
class of 1953 were least likely to be contacted and also least likely to 
to return the questionnaire when located. 

In addition, there were significant differences between schools in 
terms of percentages of returned ques tionnaires . For example, for the 
class of 1962, one school reported a return rate in the 91-100% range. 

On the other hand, for that same class, another school reported a return 














rate. in the. 11-20% range. Obviously , one school was being over-sampled 
while the other was not sampled enough. These differences between 
schools were even more striking for the classes of 1958 and 1953. 

Moreover, when these differences were. based on the number of graduates 
assumed to have been contacted, the range of differences was even gie.at.ei . 

Factors responsible for the between—school differences in the return 
rate of questionnaires included geographic regional differences, size of 
school, type of school (vocational or comprehensive), graduates’ loyalty 
to the school, type of curriculum chosen (vocational or academic), and 
race of graduate. 

This selective bias in the return rate meant that the graduates who 
returned the questionnaires represented a very different set of people 
than the. group composed of all the students who were sent the questionnaires. 

To reduce this bias Eninger employed a sample correction procedure. 
Questionnaires had not been received from two groups of graduates those 
who had never been successfully contacted (address-unknown cases) and those, 
who had been contacted but who had never returned the completed question- 
naire (nonrespondent cases). 

In order to reduce the first: source of bias, a random sample of 
address -unknown cases was selected. The research personnel then intensified 
their efforts to locate these graduates and to obtain a completed question- 
naire. from them. From these selected graduates the researchers hoped to 
get a completion rate of 90% or better. The small sample obtained this way 
could then be taken as an estimate of what the returns would have been 
like if all of those not located had been able to return questionnaires* 
Similarly, the researchers intensified their efforts to get a small, 
randomly selected group of nonrespondents to complete questionnaires. 

. in the study a 5% sampling was taken of address-unknown cases. 

This meant that if 100 vocational curriculum graduates of the class of 
1962 were not located, the investigators randomly selected five 1962 
graduates and then intensified their efforts to locate and get completed 
questionnaires from them. These five responses would then be counted 
as 100 responses on the assumption that they were typical of the address- 
unknown graduates. 
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By this means Eninger sought to reduce the bias in his follow-up 
information, lie points out, however, that this correction procedure is 
not perfect. If the randomly selected sample is not typical of the 
population it is seeking to estimate, any error or bias is multiplied 
many times. Thus, although the correction procedure tends to reduce 
bias, it probably does not eliminate it. 

Eninger also noted another source of potential error in the 
correction procedure. Whereas the uncorrected sample consisted only 
of questionnaire responses without any interaction with research per- 
sonnel, many graduates in the correction sample gave their responses 
with the aid of an interview. These two slightly different information- 
getting techniques might very well have influenced the answers received. 
For example, a graduate might have been more inclined to divulge certain 
kinds of information in an impersonal questionnaire than in an interview. 



Although these difficulties were recognized, they were impossible 
to circumvent. Eninger stated that no feasible alternate procedure exis 
Because of the problem of systematic biases both in the corrected and 
uncorrected sample data, both sets of data were presented in the report. 



In 1966, Altman and Morrison reanalyzed Eninger’ s data. Eninger ’s 
emphasis had been on evaluating the post-high school experiences of T & I 
program graduates as compared to academic program graduates. He had not 
evaluated specific high school vocational education programs nor had he 
been interested in qualitative differences between vocational programs 
given by the various schools. Altman and Morrison wanted to identify 
characteristics of successful vocational and comprehensive schools and 
to compare these with characteristics of relatively unsuccessful schools. 
They aimed especially at finding community factors and school-community 
interactions which were related to the success of the school. They 
stressed those characteristics of the interaction which could be controlled 
by the schools . 



Eninger had collected extensive follow-ups on graduates of 50 voca- 
tional and 50 comprehensive schools. The later investigators immediately 
eliminated 16 of the vocational schools and 18 of the comprehensive schools 
from their study. Twenty-seven of these schools had fewer than 15 graduates 
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in T & I each year. This 
indexes for these schools 



number was too small to provide stable placement 
Three more schools were not considered because 



most of their graduates were Negro and had relatively unsuccessful placement 
No- matching Negro schools with good placement could be found. Finally, the 
investigators eliminated four schools which were located in large metro- 
politan areas where a meaningful network of community organisations and 
school-community relationships could not be established. 



From the remaining schools , the researchers selected the eight 
vocational and the eight comprehensive programs whose graduates were 
the most successful. In a similar manner, they picked 16 relatively 
unsuccessful programs. 



As criteria for success, Altman and Morrison used four measures 
indicating the successful placement of graduates. These measures estimated 
initial placement time, the relatedness of jobs to initial training, job 
satisfaction, and job security. Each measure was computed for the 
individual graduates. The school’s score was the average score for all 
graduates from whom these data were available. Only graduates of the class 
of 1962 4 were considered. Their experiences seemed to be the most relevant 
since information about the schools and communities only began to be col- 
lected in 1965. 

Once the appropriate schools had been selected, at. least five people 
were interviewed at each school: the principal, the coordinator of the 

1 & I curriculum, the head of placement, a guidance counselor, and a 
teacher. <: 

Employers of the 1962 graduates whose questionnaires were used were 
also interviewed. In addition, local firms which might, be expected to 
employ T & I graduates were also contacted. For every trade that the 
school taught, the Employment Security Office identified potential 
employers. Then the employers were categorized with respect to the number 
of people they hired in that trade. At least one employer of large, 
medium, and small numbers of T & L graduates was interviewed. 

For every T & I course offering of the school, one labor union 
representative was interviewed. This representative was generally the. 




uruon president or business agent;* Thu appropriate union located nearest* 
to the school was selected. 

In order to choose. die moat relevant community organisations, the 
investigators made a ranked list of nationally prominent: "service , civic, 
social, and fraternal organisations whose objectives logically implied 
the possibility of influence on the placement and employment experience, 
of school graduates [p, 6]," In each community the interviewers selected 
the ten organisations which were highest: on the .'list and available in the 
community. 

In all communities the manager of the local Employment Security 
Office was interviewed. 

At this point the investigators pulled together all the material 
they had collected and used a variety of statistical and analytic technique 
to find relationships between the school-community interaction and the 
employment success of graduates. The result of this analysis was a lengthy 
consideration of the characteristics of the. schools, the characteristics 
of the community institutions, and the nature and frequency of contacts 
between the schools and their communities. 



Decker (1968) was also interested in identifying the characteristics 
of successful vocational programs; however, his investigation differed 
from that of Altman and Morrison in several important ways . First, Decker 
was concerned only with vocational programs for disadvantaged youth. His 
definition of disadvantaged incorporated definitions used by the Office of 
Economic Opportunity, the Office, of Education, Title I (ESEA) , and the 
Division of Vocational and Technical Education of the Office of Education. 
These definitions took into account: economic deprivation, educational 
background, health, cultural isolation, and lack of motivation. Second, 
Decker, unlike Altman and Morrison, did not have previously collected data 
available to him. Because he did not have this storehouse of information, 
both his selection procedures and his criteria had to differ from those 
employed by Altman and Morrison. 

Altman and Morrison had looked at 66 programs reviewed by Eninger and 
had selected 16 "good" programs, with "good" defined oil a relative, scale. 






Of those programs reviewed by Kninger, the 16 "good" programs were better 
than the 50 other programs. Thus, these "good" programs were not necessarily 
the best programs in the country, 

Decker tried to use an absolute criterion, rather than a relative one. 
First, he said that an effective program should "have been in existence 
for some period of time, . ... have been able to demonstrate reasonably well 
in about every criterion selected for study, and show a marked change and/or 
improvement in student behavior and achievement [p. 7]." Next, he searched 
for such programs. Obviously, Decker’s search techniques differed from 
those employed by Altman and Morrison, who looked at programs Eninger had 
previously selected as typical of vocational education. Decker was not 
limited to information already collected, lie reviewed more than 200 voca- 
tional education programs before selecting 40 for close study. To locate 
programs, Decker contacted State directors of vocational education, staff 
members of the U.S. Office of Education, foundation executives, and pro- 
fessional associations. Information about programs was gathered through 
mailed questionnaires, telephone queries, literature searches, interviews, 
and site visits. MDTA, Neighborhood Youth Corps, and Job Corps programs 
were excluded. 



Decker had difficulty in applying his criteria; he also pointed out 
the lack of time for the study. The investigation started on March 11, 1968, 
and in late May the schools began to close. During these two -and -a -ha 1 f 
months, the. research team had to identify and screen programs. In addition, 
seniors in the programs selected for intensive study had to be tested. The 
problem was compounded by the Easter vacation and the April disorders 
which closed schools in several of the cities studied. Many programs had 
to be. excluded because they could not provide sufficient data by June 5, 1968 

Research teams selected 40 programs to visit. To collect information 
in a stand ardised fashion, interview guides were developed. Interviewers 
collected information on the characteristics of the program, the students 
served, the enrollment procedures, and the administration of the program . 

The visiting team observed the students, facilities, and "climate" of the 
schools. 




To evaluate the programs objectively, the team looked at: placement 
statistics, dropout, rat: an , student achievement , and program costs. 

Seniors were given Stanford Achievement Tests. The school was provided with 
a data reporting form for each student in the program. The school first 
listed the student’s age, race, and sex. The form asked for the kind of 
program in which the student was enrolled and the student's scores on 
standardised tests before and after being enrolled in the program. 
Information about the student's IQ, grade-point average, and personality 
could also be included. 



After this information was collected, programs were grouped in terms 
of the students they served. These categories included inner-city dropouts, 
rural and small-city dropouts, mentally retarded youth, delinquents, and 
non-English speaking students. 



The primary criteria used by the research team referred to the pro- 
gram's impact on educational achievement, and on job training and placement. 
In order to study achievement, the team usually relied on a preprogram- 
postprogram comparison. Seniors in the graduating class of 1968 were 
studied. When preprogram achievement data were not available for this 
population, the researchers tested students in grades 10 and 11 and compared 
their scores to the scores of the 12th graders. The researchers had wanted 
to compare school-administered tests with Stanford Achievement Test results, 
but they were often unable t:o do this since many programs did not administer 



tests . 




In order to evaluate, the program's placement procedures, the researchers 
compared the achievement scores of students placed in low-skill jobs and 
students placed in high-skill jobs. The investigators were looking for a 
correlation between achievement in school and placement in a skilled job. 

The team also looked at the placement and achievement of students of 
different sexes and ethnic groups within the same program. 

A cost-benefit analysis of each program had been planned, but the 
investigators were hampered by lack of time and data. Many programs 
did not have follow-up information about the earnings of their graduates. 

Some of these programs were in their first year of operation; others had 
never conducted a follow-up. 
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In addition, cost-pcr-student information was often unavailable. 

In some, cases the investigators could not have access to this information 
because of school board disclosure policies; in other cases, the amount: 
spent for one particular project could not be determined from the avail- 
able information. ^ , 

Because of these problems , the investigators decided to compare the 
cost per pupil of programs for the disadvantaged and regular high school 
programs. This information was gathered from 7 intensively studied pro- 
grams and 32 programs not intensively studied. 

Of the 40 programs intensively studied, 30 were chosen as case 
studies to be included in the report. 
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It should be clear from the above review of related research that 
evaluations of vocational education have encountered considerable prob- 
lems. Many of the same problems are reflected in the findings and con- 
clusions of the AIR study reported on the following pages. 

The next chapter of this report deals with the methods and procedures 
employed in the AIR search. 
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METHODS AND PROCEDURES 



Selecting Progr ams for the S tudy. 

During the first months of the project, various techniques were used 
to search for leads to promising vocational programs. These techniques 
included a computer-based search of ERIC materials, a general literature 
search, consultation with experts, mail inquiries, and notices in approp- 
riate educational journals requesting program information. 

A record-keeping system was devised for identifying and describing 
the vocational programs. Three, kinds of cards were prepared for each 
program: bibliography cards, program personnel cards, and program descrip* 

t.ion cards. The bibliography cards gave, author, title, and publication 
information for program documents. The program personnel cards provided 
names and addresses of persons who could be contacted for further infor- 
mation. The program description cards provided a place to record basic 
program information. 



IM^comput e. r-b as ed search of ERIC ma terials . The most rapid, compre- 
hensive, and efficient way of identifying relevant vocational programs was 
the ERIC/DIALOG Online Retrieval System. The system's hardware consisted 
of a terminal (with a keyboard input and video display) coupled to computer 
storage containing the material to be searched. The terminal was housed 



in the ERIC Clearinghouse for Educational Media and Technology at Stanford 
University. The computer, containing an ERIC data base of more, than 
12,000 report citations (from issues of R esearch in Education through 
December 1968) was housed at Lockheed Missile's and Space Company facilities 
in the Stanford Industrial Park, Palo Alto. 



Commands were typed on the terminal keyboard in DIALOG retrieval 
language to define the nature of the search. These commands were the index 
terms used in ERIC. The computer relayed, via the terminal's video screen, 
the number of documents it had identified for each command term. For 
example, in the ERIC data base there were 765 references for the index 
term "vocational education." 



Next, the computer was told that the AIR user was interested in only 
some of those references. In this operation, called the "coordinate search 
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technique, the. computer was commanded to combine certain sets of refer--- 
ences Identified by individual index terms. The result of such a comb lira 
tion was a new, smaller set of references which fitted the combined terms 

At any point in the search, an additional feature of DIALOG allowed 
the user the option of viewing index terms alphabetically adjacent to the 
entered term* That is, a portion of the relevant page in the ERIC index 
term dictionary could be displayed on the screen. The user could scan 
the display, select terms, and expand the search to include index terms 
which had not occurred. ' to him initially. 



Through all of the procedures described above, increasingly relevant 
sets of documents were identified. Because the AIR user was in direct 
communication with the computer, results were available in seconds or 
minutes instead of days or weeks. Once the desired references were 
displayed, the computer was commanded to print out a permanent record. 

AjjjiRio nal literature searc h. Because the computer data base 
included only those documents entered in ERIC by December 1968, a search 
of the 1969 issues of Research in Education was completed. Over 570 
abstracts of documents dealing with vocational education were examined. 
Thirty of those documents referred to specific vocational programs for 
high school students, but only 20 of those programs had been evaluated. 

An examination of issues of A bstracts of Resear ch and Related Ma t eria ls 
lB_.Vogaf. ional and Techn ical Educatio n (ARM) and of Abstracts of I nstruc- 
l-ioiia l Materials in Vocatio nal and Technical Education (AIM) from 1967 to 
the present was completed. Both of these publications are specialized 
supplements to Research in Education . The AIM index is published at the 
Center for Vocational and Technical Education in Columbus, Ohio; issues 
of AIM were examined during a visit to the Center by an AIR staff member. 
The search of the ARM and AIM indexes yielded a few more high school 
vocational programs which had been evaluated. ' 

In addition, 1968 and 1969 issues of Vocational Guid ance Quarterly 
ail< i American Voca ti onal Journal were reviewed for names of the most recent 
high school vocational programs which had been evaluated. This review 



helped in identifying a few programs. Some additional programs were found 
during the review of general materials and books about research and theory 
in vocational education. 

Consu lt atio n w it h e xports. An initial AIR planning meeting was held 
during the first month of the study. At this meeting, project staff 
became acquainted with other vocational education research projects under- 
taken by AIR, At this meeting possible criteria for program selection 
were discussed, and opinions about the current: methods and problems of 
evaluating vocational programs were expressed. 

During the second month of the study, while the literature search 
continued, an AIR staff member interviewed several experts in the field 
of vocational education. A list of persons visited is presented in 
Appendix B. The main purpose of these visits was to discuss criteria for 
selecting promising vocational education programs. Also, some specific 
projects which had been evaluated were identified, and new sources of 
information were uncovered. 



Concurrently with the above visits, Dr. Robert Darling and 
Dr. Robert Barnes of the Vocational Education Section of the State 
Department of Education in Sacramento, California, consulted with the 
Project Director at the Palo Alto office of AIR. At this time, many 
avenues for finding promising vocational programs were explored. 

A two-day conference of experts in vocational education was held at 
AIR's Palo Alto office during July, the third month of the study. (See 
pages 30 to 35 for details of this conference.) While the main purpose 
of the conference was to discuss the criteria for the study, the experts 
provided names of other institutions and individuals to contact about 
specific programs. 

Ma il inquiries to admin i strative units . During the third month of 
the project, about 160 letters were sent to State Departments of Voca- 
tional Education, to State Research and Coordinating Units, to State 
Advisory Councils, and to private foundations such as the Carnegie and 
Ford Foundations. This initial mail inquiry, based on names and 
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addresses provided chiefly by the Bureau of Vocational and Technical 
Education at US OK , asked for names of promising vocational programs at 
the secondary level.. The letter was modified after the AIR conference 
with experts to include the main selection criteria: placement of 

pupils in jobs or further education. Many leads to promising vocational 
programs were received from these administrative units. 

Notices in appropriat e, -journals^ A notice was placed in several 
publications requesting persons to contact the Project Director if they 
knew of programs at the secondary level which had been successful in 
placing pupils in employment or further education. American Voca ti o nal 
Journal , S chool Sh op, P hi D e 1 1 a K a. p p an , and the V ocation al Guidance 
Quarterly were among the journals which carried this announcement. 

Mail and telephon e i nquiries t o specific, program s. With the aid of 
a Magnetic i Tape Selectric Typewriter, over 500 letters were written 
to specific programs identified by the various search techniques already 
described. The purpose of the letters was to ask program personnel to 
state whether or not they had records which would show that their program 
had increased the numbers of pupils who were placed in jobs or who con- 
tinued on to further education. Follow-up telephone calls were made to 
the personnel of specific vocational programs who did not. reply to our 
letter or who provided inadequate information. Over 250 telephone calls 
were necessary to track down such nonrespondent, s . 



Site visits . In the event that a program still looked promising 
after studying available documents and. communicating with program person- 
nel, arrangements were made for a site visit by a two-man AIR team, The 
team talked with program personnel, following a flexible routine which 
concentrated first on obtaining information on the evaluation. While at 
each site, the AIR staff searched for comparison data by contacting local 
branches of State employment offices, or any other sources suggested by 
local program personnel. A more detailed discussion of attempts to locate 
comparison or control data can be found on pages 38 to 41. 















If the evaluation of the program's success seemed reliable , the team 
would then obtain a more complete description of the program's overall 
operation. Although none of the programs visited could show that the 
primary criteria for this study had been met, complete descriptions of a 

few programs were collected as valuable background for other sections of 
this report. 









Establishing Crl ter i a 



The original criteria suggested 
would be f.or programs which had been 
lowing ways : 



ioj. this study stated, that the search 
successful in one or more of the fol- 



(a) 

(b) 



improving retention rates (i.e. 
producing measured benefits of 



> decreasing dropout rates) ; 
achievement; 



(c) assuring the employability of graduates. 

This section of the report will deal with how these 
modified and refined. 



criteria were 



During the first two months of the contract (May and June 1969), 
the three original criteria were discussed at length by AIR staff. In 
mid May, for example, in order to avoid unreliable data, "measured 
benefits of achievement" was interpreted to mean benefits of cognitive 
achievement as measured by standardized tests. The AIR staff believed 
that the most reliable data would be available from such tests. 

The original wording of (c) , "assuring employability," implied that 
a program should guarantee jobs for every one of its graduates. This 
seemed an unrealistic criterion, since placement rates are subjected to 
many influences beyond the sphere of a program itself. On the other 
hand, model programs would be expected to show positive trends in place- 
ment. "Assuring the employability of graduates" was therefore altered 
to read "increasing the employability of pupils." Subsequently, the 
phrase "or raising the proportion of pupils proceeding to further educa- 
tion was added to (c) , because expanding opportunities for further 
education in America have prompted vocational education programs to 
persuade graduates to continue their education. A model program should 
be successful in increasing the total placement rate, including those 
going into further education as well as those going into jobs. 

If a model program is to be better at placing its graduates in fur- 
ther education or employment than are other programs or institutions 
operating under similar conditions, then comparisons must be made. 
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The. AIK views of the criteria were influenced during May and June 
by meetings with officials of the Bureau of Vocational and Technical 
Education at: USOE > with staff of the Center for Studies in Vocational 
and Technical Education at Madison ? Wisconsin, and of the Center for 
Vocational and Technical Education at Columbus, Ohio, and with. Dr.- Louis 
Decker and Dr. Max Eningcr . 



In the third month of the contract (July 1969) , a conference was 
held at Palo Alto, attended by the USOE project officer, the AIR project 
staff, and five consultants prominent in the field of vocational educa- 
tion (see Appen44x--B) . At the conference, there was considerable discus- 
sion of the criteria* AIR staff pointed out that the study should not 
have criteria that would result in no programs being shown as successful, 
any more than it should have results showing all programs as successful. 
Each of' the criteria for the study was discussed in turn. Some of the 



remarks of the Project: Director should be quoted at: this point: 



" Measures of achievement : We understand that USOE would like us to 

regard standardized tests as the measures of achievement in this study. 
Standardized tests have several characteristics: they are. composed of 

items which have been carefully selected as measuring performance of 
pupils i.n' certain w ays; they have been field tested on large normal 
samples; the method of administration for each is exactly and unambig- 
uously stated; careful instructions are given for scoring them; population 
norms are normally established for them; their reliability and validity 
have been determined, and are known to be high. They include tests of 
skills , ab.i lities , achievement , and performance . 

"Our preliminary studies of vocational programs indicate that stan- 
dardized tests are not widely used in them. In many instances they are 
not considered appropriate; standardized tests at present, available 
usually measure types of educational achievement which are not central to 
the. goals of vocational programs. Perhaps they should be; after all, 
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reading ski. 3.1s am assent ini fco many courses in vocational education, But. 
unions standard;! aad tents have been uflod in n fair number of vocational 
programs » they are of little value no a criterion of success, 

"Boros' Mon in 3 Mnasurenvntn Yearbook contains many une t andar di zed 
« tests relating to vocational education* Unfortunately, few are used at 
all, and the ones that are used seem to be specific to only a small number 
of localities, usually including those of their authors! 

"No doubt there are many locally constructed tests that have been 
used but which have not reached the. pages of Buros. Some of them are 
excellent, and may grant entry to certain trades to those who perform well 
on them. Many professional and technical associations have devised their 
own examinations , .of course. 

"There are undoubtedly many direct, measures of achievement in voca- 
tional education, but all except standardized tests lack national recog- 
nition or national norms. 



"We are well aware of the shortcomings of standardized tests, partic- 
ularly for measuring achievement at the tails of the distribution, as in 
compensatory education programs. Standardized tests still represent, 
however, the best single group of predictors of scholastic performance 
which wc have. Their prediction of on-the-job performance by those who 
leave high school is not very good, but we have no better in-school 
measures. To argue at this stage that other measures should be used is a 
waste of effort, since others have not yet been developed. On the other 
hand, we should be careful not to overvalue present standardized tests so 
that funds do not become available to develop better ones. 

"If standardized tests are no t to be used as the criterion of achieve- 
ment in vocational education, what other measures are there? Certainly 
there are no other direct measures, applicable during the time a pupil is 
in the program (in school, that is) which can easily supplant the stan- 
dardized tests, 

"There are, of course, indirect measures of the program's success, 
and the other three criteria fall in this category. 
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’ ’Dr opout ra Us : Dropout. rates, huwovtr they are ca.1 coin ted, ami 

represent only one aapect: of a program's emeecsr. , and there :1s little, 
evidence that a program* r. increased holding power is closely related to 
improved performance (or groat or employability)* The. improvement of a 
dropout rate* presumably allows that the program has succeeded in intarosting 
pupils sufficiently for thorn to stay in high school instead of dropping 
out. But securing pupils’ into rant docs not guarantee that they have boon 
educated vocationally . If a program succeeds in persuading pupils that 
school is a better place than the working world } it may have merely post- 
poned their crisis of expectation. It may simply preserve them a year or 
two longer within its protective pale without equipping them better to 
face what they must meet ultimately. 



"But that is a very negative view, and there is only a little evidence 
(in the reports we hold) to support it. 

"If we assume that a reduced dropout rate, is a valid index of success, 
we must be sure that it is soundly computed, arid based on reliable data. 

"A pupil is considered as belonging to a class from the date he 
presents himself at the school until the time he permanently leaves for one 
of the causes recognized as sufficient by the State. The. date of permanent 
withdrawal is the ’date on which it is officially known that the pupil has 
left school, and not necessarily the first day after the date of last 
attendance.’ This definition is open to abuse, since officials can deny 
knowledge of a pupil’s departure, for months after he has really left, thus 
improving the dropout rate. 

"USOE also lists 19 reasons for dropping out: physical illness, 

physical disability, mental illness, mental disability, behavioral diffi- 
culty, academic difficulty, lack of appropriate curriculum, poor pupil- 
staff relationships , poor relationships with fellow pupils, dislike of 
school experiences, parental influence, need at home, economic reasons,, 
employment, marriage, pregnancy, other known reasons, unknown reasons , new 
residence (school status unknown) . (Death of pupil is listed separately, 
not being a reason for dropping out.) Each of the 19 i.s carefully defined. 




"This van#?. • of; roaeoua raises some serious quesHoms, Many of the 
reasons for dropping out: a to only partly connected with schooling j others 
have no connection at. all, A program which falls to improve a dropout 
rate may have had to counter a sharp increase in jobs available to dropouts 
in the area, Another may have witnessed a general exodus of people from 
the school district, 

"Thus changes in dropout rates need to be viewed against the back- 
ground of what has happened in a school district during the lifetime of a 
program. Such details are not always included in the reports, and may be 
hard to obtain, 

"Employ ability: This criterion seems to be the obvious one — the 

utilitarian one by which vocational programs should be judged. Can grad- 
uates of a given vocational program succeed in the world of work? 

"Unfortunately, we are still faced with the .question of what comprises 
success. Immediate employment? In what kind of job? How long retained 
in the job? Was the job related to the vocational training the pupils 
received? (Job advancement and acceptance, of increased responsibility 
might also be considered.) 

"What records of employment are available? What of graduates who 
cannot be traced? Is the local sample available likely to be a biased 

one? 

"E ligibility for fu rther educati on: Eligibility in this context may 

mean qualification or acceptance. We are talking about eligibility for, 
or actual enrollment in, further education. , 

Qualification: The Regents examination in New York State, or 

32 th grade completion certificates in many districts, 
or other paper certifications which are gained by 
high schoolers before proceeding to college, might 
be used as the criterion for qualification. The 
members in, a program gaining qualifications would 
be counted. 
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He*, re there wight be no quail ification gained in some 
cases , but graduates would be admitted to institu- 
tions of higher or further education . Acceptance 
would be. the criterion. 



"The typo of further education provided might be Important. What 
qualified a pupil to be. scored in this category? Part-time further educa- 



tion? Full-time? Episodic? Is formal registration enough or must at ten- 
dance. records bo examined? Just as for employability, what .is the state 
of the. records likely to bo available.?’ 1 



In the general discussion of criteria at the. conference, .it was 
emphasised that success was relative, and that a program had to be con- 
sidered in relation to its context for one to judge properly its success. 



At the conference it was agreed that the ultimate criteria of success 
for vocational programs were: (a) placement in jobs related to the voca- 

tional training provided; and (b) enrollment in further education courses 
related to the vocational training provided, it was acknowledged , however, 
that placement in other jobs, and enrollment in other kinds of further 
education courses, should be considered as criteria too, since the overall 
aims of programs were to increase general employability and participation 
in further education. It was agreed that the primary criteria (ones that 
must be met by model programs) should be that the program examined had 
increased: 



(a) placement in jobs'; and 

(b) placement in further education. 

Soma programs might induce greater increases in (a) than in (b) , and 
vice versa. The AIR staff decided to use the term total pla cem^t_rate. 



for the overall percentage of graduates placed. Later, it was found that 
besides those placed in further education or in jobs there were some 
graduates who had gone into military service or had become homemakers, 
and a very few' who were, unemployed. Because the search was restricted to 
program gra duates , that, is, students w'ho had completed twelfth grade, not. 
many cases of unemployment were likely, a fact of importance later in 
the study. 
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Thcs secondary criteria (ones that did not have to bo wet by model 
programs) were that the program examined had 5 

(c) improved attendance and retention rates; and 

(d) improved scores on standardised academic achievement tests, 

The task of establishing v/hat constituted an increase in total place- 
ment [under criteria (a) and (b) above] was left to AIR, The project staff 
realised that total placement rates were vary much open to the influence 
of local factors. For example, if the major employer in a small city 
opened a new factory during the life of the program, the total placement 
rate would be affected, and the program could scarcely claim full credit. 
Similarly, if a local junior college closed down for lack of funds, or 
dropped some of its courses, the total placement rate would be affected. 

Ideally, it would be desirable to conduct a follow-up of graduates 
that would yield a 90% or greater return of replies, showing the total 
placement rates for both a program group and a suitable, contemporary 
comparison group. These might be termed the "raw” rates. The only 
independent variable in the evaluation design would be the treatment 
available to program and nonprogram students. Program students would be 
’'matched" with nonprogram students on such variables as socioeconomic 
background, age, sex, and scholastic achievement. In such an ideal situ- 
ation, a simple but appreciable (say not less than 5%) difference in the 
raw rates for the two groups, in favor of the program group, would be 



acceptable, evidence of success. 

It was supposed that increases in total .placement rate might be 
detected in four ways : 



(1) The ideal, experimental-control comparison (already described). 
An example, is shown in Figure. 1. It should be noted that three 
different "classes" arc included in the. graph in Figure 1. Of 
course, if the. class varied from year to year, those variations 
should influence both program and control groups equally. 

Figure 1 shows the program succeeding in all three years, 
increasingly so, • 
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Fig. 1, Evaluation Type X. 

(2) The comparison between, a program group and a similar group 
which graduated before the program began. Here is a case of 
different "classes” in different years being compared, a much 
* more difficult task than the first kind of comparison repre- 
sented. An example of the "raw" rates is shown in Figure 2. 
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Fig. 2. Evaluation Type 2. 
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Such raw race a might be adj rated by factor;; shown in Figure 3, 
if any way existed to compute Che exact influence, of each of 
these factors upon the varioxio graduating classes . To undertake 
such computation, it would be necessary first to have normative 
data, and second to assign vroighU; . A search for normative, data 
was indeed made by the project staff, as will be explained below. 



The comparison between placement rates for the whole high s chool 
during the program and before it. Again external factors that 
might have changed over time have to he taken into account if 
possible. This type, of comparison raises even more problems 
than (1) and (2), however, since there may also have been 
factors operating differently on the nonprogram portion of the 
high school. Furthermore, the nature of the high school intake 
may have changed. Adjusted rates are essential if this type of 
comparison is to be made. 
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I*ig. 3. Hypothetical relationships- --between program 
total placement rate and other factors external to program. 
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The covapaUson between 
over a period of years 
have occurred. Here it 



placement rates for program students o nly > 
during which major changes in the. program 
is assumed that the student body involved 



remains similar from year to year, so that any increase can be 
attributed to the program changes. External factors might be 
t: ak to a in t o acco un t t h to u gl i "a d j us t me nt . " 



T he Sea rch f or N orm ati ve bat a 

Except for the first one, the kinds of comparisons that have been 
suggested in order to show increases in total placement rate require nor-' 
mative data as a basis for adjusting the rates. Adjustment is essential 
because influences on programs fluctuate over time* 

This section of the report will examine the nature and quality of 
normative information, and its availability at the Federal, State, and 
local levels. 

In the Department of Labor, the AIR staff contacted Manpower Programs, 
the Bureau of Labor Statistics, and tbe Office of Information Publications 
and Reports. The Bureau of Labor Statistics furnished several Monthly 
Labor Review reprints concerned with the employment of youth, and sug- 
gested that staff members visit the Government Documents Section of the. 
Stanford University Library. Based on an examination of the documents 
found at Stanford, and those received from the Bureau of Labor Statistics, 
the AIR staff concluded that Federal norms on employment of high school 
graduates were not available. The most suitable document was the Special 
Labor Force Report Mo. 100, based on the current Population Survey con- 
ducted by the Bureau of the Census. Though covering the employment of 
high school graduates and dropouts in 1967, this report includes persons 
from 16 to 24 years of age, which extended beyond the age limitations of 
the study* The other national agencies surveyed through the Stanford 
facilities and during later telephone contacts included the San Francisco 
Regional Office of the U.S, Office of Education, the National Center for 
Educational Statistics, the current Population Survey, the Social Security 
Administration, the Bureau of the Census, and the U*S. Training and 
Employment Service. 







Similar steps were, taken to survey possible sources of normative 
data at the State and county levels * The survey of documents at Stanford 
University revealed that several States publish statistical yearbooks 
which cover the number and sex of high school graduates, but none deal 
specifically with employment rates immediately; after graduation from 
high school. 



Next, the search was intensified by telephoning State agencies, 
using California as a model. Several agencies within the California 
Department of Employment were contacted, including Employment Service 
Information, the Director of the Youth Division, and the Library and Labor 



Statistics division of the Department of Industrial Relations. Although 
these agencies are concerned with employment among high school age youth, 
no data were available specifically on the employability of high school 



graduates . 



Similarly, sections within the California Department of Education 
were contacted, such as Reference Services, Educational Research, Occupa- 
tional Preparation, and the Coordinating Council of Higher Education. 

The results were again negative, since these agencies focus mainly on 
high school graduates continuing on to college after graduation. 

The contacts made at the State level proved profitable, however, in 
that several school district studies and private research organizations 
were mentioned. Sources specifically referred to were: the Carnegie 

Commission in Berkeley; the San Mateo Career Information Service; the 
Arthur D. Little & Company in San Francisco; the METRO Study in Pitts- 
burgh; the Department: of Industrial Relations at the University of 
California, Berkeley J and guidance and research centers in local school 
districts . 

As the search continued in this direction, guidance and research 
centers appeared to be the. best source of the type of employability data 
needed in the AIR study. Guidance and research centers in some high 
school districts had systematically followed graduating classes, usually 
over a period of from two to five years. However, this information could 
not be generalized beyond the school district, due to the uniqueness of 









community population and industrial characteristics * Obviously, data 
from a follow-up of high school graduates in a middle-class suburban 
community could not; be reliably used as a basis of comparison tor grad- 
uates from lower-class urban high schools. 

Since the follow-up of the other private corporations and research 
sources produced negative results, the AIR staff concluded that normative 
data were, not available at cither the Federal or State levels , and that 
the data which existed at the local level did not allow generalisation. 

Next, the AIR staff reviewed the research literature in vocational 
education and contacted by mail and telephone several investigators who 
had previously conducted national evaluations of vocational education. 

The most relevant single, study found was Eninger (1965, 1968), which is 
reviewed in an e a r .1 i e r section of this report. Eninger (1968) summarised the 
importance of his study by saying that "it provides a benchmark of data 
against which changes in vocational education can be measured in the 
future [Ch. 1, p. 1]." While his work would appear to fit the normative 
needs of the AIR study, several points must be considered. 



First of all, the last graduates included in the Eninger study were 
from the class of 1962, and the 1963 Vocational Education Act prompted 
many changes in vocational education. Since the AIR study was restricted 
to vocational education programs between the years 1964 to 1969, data 
obtained prior to 1964 could not be used. 

Second, Eninger’ s data covered the entire United States, and could 
only be compared to another study of national scope. Consequently, this 
normative information could not serve as a basis of comparison for single 
vocational programs , which were the concern of the AIR study. 



The. other studies reviewed, including Kaufman, Schaefer, Lewis, 
Stevens, and House (1967)-, Kaufman and Brown (1968), and Altman and 
Morrison (1966), did not contain normative information relevant to the 
AIR study. Although Kaufman et al. (1967) reviewed employment trends for 
different size communities , this information was acquired from the 1960 
census, which was too old for our study. 
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Other mail and telephone contacts 
The U.S. Office, of Education forwarded 
of vocational completions during 1968, 
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also produced negative results, 
summary tables showing the status 
but these data could not be adjusted 
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in terras of contemporary employment influences. 



Eninger informed the project 
being collected for Project METRO 



staff that normative information is 
which is an investigation of vocational 



graduates in 13 American cities. Unfortunately, this information was not 
available in time, and there is some doubt whether it would have suited 
the purposes of the AIR study. 

Other personal contacts produced similar results--- no normative infor- 
mation useful to this study could be located. 



Upon completion of the intensive phase of the normative search, the 
AIR staff tentatively concluded that no reliable comparison information 
could be obtained from Federal or State sources, or from private research 
organizations or individuals. The likelihood of finding such data locally 
was still considered significant, however, and the AIR staff continued to 
investigate leads on normative information as the survey of vocational 
programs continued. 

Using the criteria, by then well established, the AIR staff began 
site visits in September, and by the end of the month had completed visits 
or calls to a wide diversity of vocational programs. Not one of the. 
programs contacted seemed likely to be able to meet the criteria, for 
reasons discussed under Results later in this report. The Project Officer 
at USOE, when questioned, confirmed that he wished AIR to proceed using 
the same criteria. The AIR staff agreed to do so, but indicated that they 
would both intensify and extend the search on account of the apparent 
paucity of suitable programs. 
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One of the original aim:: of this study was t he ckuicri.pt ion of those 
pro (•ramw chat* met the. criteria for the study. It wan in fended that pro- 
grams that had been successful .in Increasing the numbers of pupils placed 
in employment or further education should become models for others to 
Imitate or re placate. 

No programs were found that could meet the criteria , hence none will 
be described. Such a result needs some explanation: it says something 

not only about the programs at present operating in the United States but 
also about the criteria and the difficulties likely to be experienced by 
evaluators in meeting them. 



First-hand investigation of some 30 programs and examination of the 
evaluations of several hundred others gave the AIR project staff oppor- 
tunities to identify problems of establishing programs so that they can 
be properly evaluated, and opportunities to analyse problems of evalua- 
tion itself, In this section of the report, the programs studied will 
be discussed, and publications in which attempts to meet the criteria 
were reported will be reviewed. 



Th e Pro g r ams S t u d 1 e d 

Of the 445 vocational education programs contacted during the search, 
a large proportion ( 43 %) did not have, comprehensive follow-up information 
on their graduates. That is to say, 43% of the programs had not found 
out what 75% or more of their graduates were, doing. (The level of 75% 
was selected arbitrarily as representing a comprehensive follow-up.) 

Some of the programs in this category had not operated long enough to have 
follow-up records. Others had follow-up records for one year or several, 
but in many cases follow-up was done quite informally when information was 
volunteered by graduates returning to visit: instructors. Elsewhere , grad- 
uates were asked to complete and return questionnaires that had been mailed 
to them, but insufficient data were obtained about nonrespondents. 
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Programs (hat did nni. have eomprehoi »ci' vc* follow up info* mat:lon were 
hoc investigated further. Ar. wer. pointed out: ear.1 ier in this rpputffc , 
when ■ fo.n.ow-up yields a very in<;omp.let;e rot: urn, it: :La very .liberty that 
per sou a* who do respond are u special group (a»g*» better jobs, no com- 
plaints) and not ropresenttitivo of all persons who were in the program 
at the same time. 



Comprehensive follow-up information (from more than 73% of the pro- 
gram graduates) was avoidable from 34% of the programs contacted* For 
a large number of the programs in this category, response rates were 
over 90%. These programs may not; have been represent ative of all voca- 
tional programs, however. For example, conditions under which many of 
these programs operated facilitated attempts to locate graduates and 
obtain requested information, Such conditions included relatively small, 
stable populations in places where location of graduates was known, and 
where good employment and educational opportunities existed so that grad- 



uates did not have to seek these outside the community. 

The programs in this category, that is, those with comprehensive 
follow-up records, seemed to offer the best chance of detecting trends, 
including increases, in the total placement rate (placement in employment 
or further education) , But in no cases were trends detectable because 
virtually no graduates were unemployed. That: is, either the graduates 
were in the armed forces, were married and had become homemakers, or, of 
the remainder available for work , all had been able t:o secure jobs or 
to continue their education. Before dropping such programs from further 
study, however, the AIR project staff asked the program personnel if 
increases in placement could be shown by comparing the graduates of the 
vocational education programs with similar graduates who were not in the 
programs* In no case was comprehensive follow-up information available 
on an appropriate sample of nonprogram graduates. 

The balance (23%) of the programs contacted were not studied either 
because their funds were from labor-oriented sources, or because they 
were found not to be at: the secondary level, or because they were ter- 
minated prior to 1964, or because, they had fewer than 25 graduates, or 
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bouauuc- t Ik* JL ir compirohcnuiva followup roeordu h.*»d not boon dimly sad yoL 
by program personnel , 

The problom of doU’miirlng whether or not a program mat the search 
criteria ultimately hinged on the absence of any guilt. able baseline against 
. which to measure the effectiveness of the* program . Without, baseline 
information 3.1 ike that provided by appropriate, comparison groups or suit** 
able national, State, and local normative date, evaluation information 
on vocational education programs is severely limited, (See pages 38 to 41 
for discussions related to this point.) 



Problems in Meeting the Criteria 



It should be. noted that a great many of the program personnel con- 
tacted during the Mil search were not at all on the defensive, so far as 
the MR staff could judge, when asked about the graduates of their pro- 



grams. Many of them pointed out that they were required to make some 
kind of report to their respective State departments of education, 



usually ’in terms o* numbers of graduates gaining employment in occupa- 
tions related and unrelated to the vocational education provided. The 



program personnel were ready to supply such figures, whenever they had 
computed them. In some cases, where the figures had not been computed, 



they offered to compute them for AIR. 



But v;hen the program personnel were asked whether they could show 
that their programs bad increased placement in employment or further 
education, they were both quick to understand the question and to say 
that they wished they had data to show such increases. In other words, 
the atmosphere among local vocational education directors seemed to be 
one conducive to follow— up types of evaluation and the directors seemed 
open to suggestions about how it might be done. It was equally 
apparent, however, that very few of them had considered evaluation as 
requiring a comparison, between program and nonprogram graduates* 

The ME project staff became more hopeful in their search when they 
encountered isolated programs in which there had been attempts to meet 
the criteria. Several of these should be. summarised here to indicate 
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tit*,' types oi prub.ii f* tlmt prevented those program t and others front bo in g 
dia.rnct-crir.uU an , We<v ? :.mfu1 " by the A'Ul study, 

Ju a vocal 1 oanl educate er» project la Illinois, the director knew 
before the prog cam began that there would ho about ,100 pupils dropping 
out that year 3, and ho selected 50 potential dropouts for the program,* 
lie expected to he able to select another 50 for comparison purposes , but 
could not do so because of lack of coop oration from high school faculty. 

3,n the second year of the. program, ho had difficulty in securing pupils 
for the program because a nearby college instituted a high-status program 
for culturally deprived youth, The college program had the effect of 
Inducing pupils to Ignore the vocational program and to favor the college 
preparatory program* By the third year all possible "comparison" pupils 
had been absorbed into the program. 

In another vocational project, this time in Missouri, a total of 
400 pupils were selected and divided randomly into control and experi- 
mental groups. Parents of pupils selected for the experimental group 
did not in all cases give permission for their children to be in the 
program, however, and bias was introduced into both experimental and 
control groups by this factor. No fewer than 46 pupils had to be dropped 
from the experimental sampling and replaced by others from the control 
group. During the program, 124 pupils were lost from the experimental group 

In a Kentucky program, pilot and control classes were selected in a 
number of schools, but pretesting revealed rather large differences in 
several subject areas between the two groups of classes. No other 
classes could be selected as controls. 

A Minnesota program serving 531 pupils employed an experimental- 
control design, but there were, only 21 pupils in each group. The groups 
were matched on sex, age, last school attended, and date of registration 
for the program, but there was no evidence that they had been randomly 
selected. The analysis based on these two groups showed that the program 
had had no effect on total placement rate, 

A city in Oklahoma conducted follow-up of all its high school grad- 
uates, thereby providing a basis for comparing vocational and other 

46 



o 

ERIC 















graduates, but did not draw any compnr J a on because the vocational r.lu- 
dcnu a were not it 1 cm tilled as a separate group. 

In Callfo ruin , one program director expressed interest In die. AIK 
criteria and wen I to conn! dor able trouble to compare program and non- 

program graduates. The work experience program placed 72 pupils (out of: 224 
applicants) in jobs while they were still in secondary school. The evalua- 
tion consisted of follow-up of these. 72 pupils and of another 72 chosen at 
random from the other applicants who had failed to get work experience* 

In this case, a bias operated against: the comparison group* since they 
had not been selected while in school for work, experience, presumably 
they were inferior in some way to those chosen, hence they were not an 
adequate, group for comparison. They were the rejects. 

In Connecticut, the director of a program wished to establish a 
control group but was not permitted to do so by community pressure. 

The community demanded that all eligible pupils be admitted to the 
program. 

In Wisconsin, some programs had fol-lowed-up all their graduates and 
there had been follow-up of nonprogram graduates. For both groups, the 
total placement rate was close to 100%. The program directors reported 
in several instances that even nongraduates (dropouts) were employed the 
moment they left school, because of the high local demand for labor. 

The examples quoted above are not typical of the 445 programs studied 
by AIR. In most of the 445, little or no attempt was made to select any 
kind of comparison group for follow-up. The criteria and evaluation designs 
actually used by vocational educators were, clearly different, at the time of 
this study, from criteria set by people outside vocational education, and 
different from those established for the study. The next chapter discusses 
the differences, and suggests possible courses of action. 
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The old of outcome of tins study by AIR waft that none of the, 445 pro* 
grams studied could be shown clearly to have increased the placement of 
graduates in employment or further education (total placement rate). 



The reasons wliy none of the programs could show increases in total 
placement rate wore related far more closely to problems of data collection 



and evaluation design than to merits or shortcomings of the programs 
themselves . 



Where data collection had been reasonably comprehensive, valid com- 
parisons were still almost impossible to make. Comparisons between 
different groups at different times, or between the same group at different 
times, were invalidated by factors outside the programs, such as fluctuations 
in the local employment situation or the availability of college education. 
Comparisons of contemporaneous groups were usually invalidated by the groups 
being too dissimilar, for a variety of reasons, as shown in the last chapter. 



In the few instances where valid comparisons could be drawn, no 
increases were found because almost all high school graduates were placed 
in employment or further education regardless of whether they received 
vocational, academic, or general education. 

In other words, the combined criteria of placement of graduates in 
employment or further education are apparently not very useful in deter- 
mining the success of a vocational program, however close placement may be 
to the objectives of vocational educators. The strictly quantitative 
criteria used in this study must be modified. , 

Suitable . Criteria for Vocational Education 

Thus the question must be asked: What are suitable criteria for judging 

vocational education? The word "suitable"' implies- that value-judgments must 
be made about what does or does not appeal to various authorities. 



There is much evidence that vocational educators have in general 
employed criteria which gain little acceptance among people who want to 
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he shown the greater bane Cl U? that vocational oclucai i on la claimed to 
provide* Thane critics? point out Clint: vocational, court? os coat coiisl dur- 
ably more than conventional education, yet there is no hard evidence, 
from 30 yearn of vocational education In the United Staten that conven- 
tional' instruction could not have been at least as beneficial* 

The. claims made by p ropon ents of vocational education can be exam- 
ined and from them ’’suitable’ 1 criteria can be drawn. The claims of the 
oppo nent s of vocational education can be examined similarly, and criteria 
can be drawn from them too. These criteria will be seen as ’’suitable” 
by the opponents. If there is any common ground between the two groups, 
if there are any criteria on which they are likely to agree, then every 
attempt should be. made to conduct studies around those criteria* 

The views of the proponents of vocational education are exemplified 
by statements in the General Report of the Advisory Council on Vocational 
Education to the U.S. Office of Education (Advisory Council, 1968). In 
its report, the Council defines vocational education as ’’the educational 
content and process through which one learns to become a competent 
worker [p. xix]." The Council says that vocational education’s ’’primary 
responsibility is to help people enter the world of work or to make 
progress in it, t:o their best advantage and that of society [p. xxi].” 

And that "vocational education will provide the hard-core essentials 
which will make it possible for him to find employment in a number of 
specific jobs related to the area of his vocational employment [p. xxii] ." 

In view of these statements of the Council, it would be reasonable 
to suppose that the following criteria would be acceptable to the pro- 
ponents of vocational education: 

(1) That vocational education will produce workers who are more 
competent than comparable graduates educated under other 
programs. 

(2) That vocational education will enable its students to enter 
the world of work more easily than comparable graduates from 
other programs. 

(3) That vocational education graduates will make better progress 
in the. world of work than comparable graduates of other 
programs . 
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(A) That: vocational education graduates will be equipped to find 
employee.!:!: in jobs related to their training more easily than 
compm abJ o graduates of other programs. 

The report of the Council contains lengthy excerpts from a review 
of vocational education made by a Panel of Consultants in 196.1-62, As 
of that date* very few studies had offered any information about such 
criteria as those drawn from the Council’s report. The Panel stated that 
"limited data are available on placement of graduates, from scattered 
surveys made on different bases, with consequent .lack of comparability. 
Few data are available on earnings of vocational education graduates 
[Advisory Council, .1968, p. A] . " 




The literature review of related research (see pages 5 to 21 of this 
report) indicated the paucity of studies since 1962 that attempted to 
measure program success by one or more of the four criteria above. 

The proponents of vocational education would be sorely pressed to 
show from existing data that any of the four criteria were being met, 
even by an individual program, It is true that in a number of States 
statistics are now collected indicating the vocational direction taken 
by graduates of vocational programs. From these statistics, judgments 
can be made about the proportion of graduates each year who proceed to 
jobs related to their training. Such figures might be thought to 
indicate that the fourth criterion listed above was being met, but the 
figures do not do so because there are no equivalent statistics for 
nonvocational graduates. All the figures can show is that the vocational 
programs had some influence on the job choices of pupils. The exact 
nature of this influence is hard to determine because the figures are 
derived from very broad job categories, i.e., a graduate from a trade 
and industrial program has a vast range of jobs that he may choose from 
that are classified as "related" to his program. Moreover, it is not 
necessarily to the credit of vocational education that it influences a 
high proportion of its graduates to enter and remain in jobs related 
to the training provided; the same graduates might have done better by 
changing fields. (Such a criticism applies to any form of specialized 
training, of course.) 
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When evaluations of individual programs arc examined , the lack of 
reliable evidence* about the effectiveness of vocational education is most 
noticeable. The reason for this lack of evidence, has lain in the approach 
of vocational educators to evaluation. Essentially , their approach scru- 
tinizes the activities of the program. Effects of these activities on 
graduates of the program are usually not studied , or, where they are 
evaluated, no basis for comparison is provided. 



As recently as 1967, one State Department of Public Instruction 
issued a set of evaluative criteria for vocational technical programs 
(Reynolds, Grobman, & McGee, 1967) that reflected well the standpoint of 
many proponents of vocational education, since it was reviewed by a large 
group of vocational educators. The chief procedure for evaluation sug- 
gested in this publication was that an evaluation team should visit the 
program and make judgments based on the criteria included in the set. 



The set had eleven subsets: all referred to aspects of what occurred 

during the program’s operation, such as its administration, the school 
plant employed, the school library available, the student activities, and 
guidance services, as well as six broad areas of vocational education 
(e.g., business education). Obviously the authors of this set: of criteria 
did not intend to evaluate what happened to graduates afte r they had been 
in the program. Their approach was close to that used by accreditation 
teams, and similar t:o those suggested by other sets of evaluative criteria 
developed for various areas of vocational education by other bodies (see 
By ram & McKinney, 1968, p. 3, for a list). Their evaluations could not 
possibly provide objective data (or soundly based recommendations) derived 
from follow-up studies. 

A more balanced model for evaluation, incorporating both assessment 
of in -school practice and evaluation of graduates in the world of work, 
was offered in Byram and McKinney’s (1968) handbook. One section of the 
handbook dealt with follow-up studies in considerable detail. What was 
missing from this publication, however, was any mention of the need to 
establish and follow-up a suitable comparison group. The type of follow- 
up suggested by By ram and McKinney would yield no information about the 
effectiveness of vocational education compared with other courses of 




instruction, although the handbook, if used, would result: In very muc.li 
better evaluation than that commonly done at present. 



The oppon e nts of voc 
achieve its declared goal 



a t ion a 1 e cl 1 1 c a t: i on 
k> D tucker (1968) 



do not: believe, that it: can 
asserts that "the skills 



that ’vocational education * teaches are. 
skills of yesterday. The one thing that 
time the students graduate into jobs . . 



obsolete. They are the craft 
is predictable is that by the 
. they will no longer be done 



the way we are teaching these crafts in our vocational schools [p. 297]." 
Others claim that the increased enrollment in vocational education is 
no indication of its excellence (without suggesting why) . They point 
to evidence in many communities that vocational education enjoys less 
prestige than college preparatory courses. They declare that children 
should not he. shunted into vocational education courses when these 
children may indeed be. capable of college work (which they assume to 
be more desirable.) . They assert that vocational education does not keep 
up-to-date, that obsolete equipment is used and procedures are taught 
that have been superseded by automation. They cite evidence that money 
spent on placement and guidance services achieves better results for 
their children than vocational education. In fact, the high cost of 
vocational education is the chief cause for attack; the opponents vir- 
tually demand cost-benefit studies. 

Cost-benefit studies could be designed to compare the costs and 
benefits of one vocational program with those of . another vocational pro- 
gram. But that is not. the. kind of cost-benefit study that is being 
demanded. Rather, the costs and benefits of one vocational program 
should be compared .with those, of nonvocational instruction. A prereq- 
uisite of such a cost-benefit study would be that benefits had in fact 
been derived from both courses of instruction. The present dilemma Is 
that there is little or no evidence of benefits from vocational education 
in terms of the quantitative criteria adopted for this study. 

There is scarcely any evidence that vocational education programs 
have had an impact, as measured by lifetime earnings or any other quan- 
titative index. The reason why this is so is that the required 








comparisons have no l: yet been drawn between vocational! programs and other 
types of education. 

The common ground that apparently exists betv;een proponents and 
opponents of vocational education regarding criteria Is that vocational 
education programs should benefit students more than nonvoca tional pro- 
grams > and that the benefit should become, visible in the students* careers 
after they leave the programs. Neither proponents nor opponents of voca- 
tional education can "prove" their arguments because of the lack of 
reliable data. In other words, the fundamental, question has to be made 
more specific: Can we fi nd out whether vocational education benefits 

students more than nonvoca tional education after they have finished their 
schooling? 

Eval u at ion Using Suita bl e C r iter la 

The most simplistic view of evaluation in vocational education is one 
which considers each program to be isolated from its environment. A pro- 
gram is then thought of as a closed system, in a state of equilibrium, in 
which teachers interact with students to produce trained students. No credit 
or blame is attached to factors such as the national economy that operate 
outside (but upon) the system. This simplistic view is quite inadequate, if 

i 

the question is whether vocational education benefits students after they 
finish it. 

It is necessary to consider each program as an open system, maintaining 
itself through continuous inflow and outflow and never in a state of equilib- 
rium. As many measures as possible must be taken of inflow and outflow, but 
the constantly changing interactions between components within the system are 
so complex that it is scarcely possible to track them. 

To speak in practical terms, inflow for a program must be described as 
carefully as possible by identifying the students’ and teachers’ character- 
istics, by noting the context of the program, and by analyzing the procedures, 
equipment, and techniques employed. Outflow for a program must be similarly 
gauged: again students’ and teachers’ characteristics have to be identified, 

to see how they have changed. But for vocational education the gauging must 
occur some time after schooling has been completed. 
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Dyer (1969) calls his model of educaUiann.l evaluation the "< student 
change” model, and ii* has similar - ] lies to the one discussed above. He 
restricts Input and output to student, characteristics , . however, since 
he is looking only at student, change » The "surrounding •conditions” 
(context) he identifies in terms of home, school, and community, Obser- 
vable activities in the school system he terms the "educational process." 
Dyer emphasizes, however, that the use of his model (described in outline 
in his 1969 paper) will only provide some useful hypotheses for upgrading 
schools, but he suggests that there is no time to waste actually testing 
these hypotheses. Instead he would try to persuade the school that; was 
"worst" according to his analysis to adopt ideas from the school that 
was "best." Such adoption might begin to close the gap between the two 
schools. Dyer restricts his examples to ones drawn from conventional 
schools, but his model could be adapted to vocational education provided 
dependable follow-up data were available. 



Analysis of inflow and outflow was what the AIR project team had 
expected to be able to undertake for at least a few vocational education 
programs. But with outflow defined strictly in terms of the total place- 
ment rate in further education or employment, no analysis was feasible, 
as has been indicated. The only instance (that was encountered during 
this study) of a plan for program evaluation that would permit such 
analysis was one prepared by Green (1969) for use by New York area occu- 
pational centers. Of the programs that were site-visited, one had 
originally had the opportunity to set up similar, contemporaneous program 
and comparison groups, but had not done so. 

Even if the ideal situation existed, in which there were -comparable 
contemporaneous groups and in which comprehensive, follow-up of graduates 
had been conducted, would the total placement rate then be useful in 
determining whether vocations], education had caused increases? Probably 
not. The problem of very high rates (and hence little difference between 
them) would still be present. Perhaps 95% of the vocational group and 
94.5% of the comparison group would be placed. Such raw rates tell too 

little about the respective programs. What is needed is a qualitative 

\ 

approach to supplement the quantitative* \ 

% 

N. i( v 



54 



I 



Instead of follow-up of the kind that is (son orally undertaken at 
present;, more complex follow-up is necessary to identify more exactly 
what happens to program and nonprogram graduates. The total placement 
rate is the sledgcdiammcr being used to crack the. nut; it is too coarse 



a measure of what is to be. assessed. 

It is true that: some studies have made attempts to follow graduates’ 
careers in detail. Unfortunately, they have had low response to their 
inquiries. Yet follow-up that is both comprehensive and. fine-grained 
seems to be the only technique that can supply the required data. 

Even if a properly randomized control group is not politically or 
ethically acceptable, the evaluation plan should still provide for some 
sort of detailed comparison of program and nonprogram graduates. For 
example, with the cooperation of guidance personnel at the high schools, 
local employers, and employment agencies, it may be possible to identify 
those locales where most of the program graduates find work. For each 
of those locales it; might be feasible to determine for, say, 18-year olds 
who took a general high school course and who bad jio specific vocational 
training before graduation, rates of initial job placement, rates of 
advancement on the job, job competency, and job retention. The same 
computations could be made for the graduates of the special vocational 
program. If these computations could be made for both groups in each 
selected locale at regular intervals, say every 10 months for 40 months, 
the trends for both groups could be compared. The advantage of this 
design is that fluctuations in the local labor market and other factors 
outside the program (which may be inherently unstable, over time) will 
have a chance to act on both groups in each of the selected locales. 

If comprehensive, and fine-grained follow-up could be accomplished, 
far more detailed conclusions could be drawn about a program’s relative 
success. Assume, for example, that the follow-up revealed that vocational 
graduates from a particular program were obtaining jobs more easily than 
other graduates, or that they were mostly entering jobs that paid well 
initially but had very limited long-term prospects. The follow-up would 
indicate well the success of the vocational program, in terms of how its 
graduates behaved after they had left: it* 
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In nummary , (die shortcomings of vocational education program ova Xu a 
Colons have boon of two kinds, when it is. a question of showing that 
vocational program:-; benefit chair graduates more, than other courses of 
instruction. • First, there has been inadequate follow-up of graduates * 
Stress has been laid upon finding out what proportion of graduates are. 
in occupations related to the vocational training provided, rather than 
upon collecting quantitative and qualitative, data about the nature of 
the graduates’ occupations. Second, there has been insufficient effort 



made towards designing evaluations so that contemporaneous program and 
(appropriate) nonprogram groups of graduates can be followed up, This 
is not to say that every graduate of every vocational program, and a 
corresponding number of nonv oca t Iona 1 graduates, should be followed. 
Random sampling, both of programs and of students within programs, 



could be used to reduce the task to manageable proportions. 
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The-' findings of, this study cannot them selves resolve the question: 
Do vocational program students benefit more after they have left the 
programs than if they had been in other types of programs? Yet this 
criterion does seem to be the one that should be applied, not only 
according to the opponents of vocational education but: also according 

to its proponents . 

If meaningful analyses are to be conducted in the future, and if 
realistic answers are to be sought, program evaluation designs in 
vocational education will have to be changed considerably. 

There are two recommendations that can be derived from this study 



(1) That funds should be set aside for intensive follow-up of 
the graduates of selected, promising vocational programs, 
and for similar follow-up of a suitable comparison group 
of nonvocational graduates for each program selected. 

(2) That assistance should be provided to local vocational 

• program evaluators in the form of training in evaluation 
procedures and design. 
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At. the local school district: level there are many different approaches 
to evaluation , and not all of these evaluative efforts are addressed to 
follow-up of the program graduates. The reports of these, evaluative 
efforts are typically brief and selective in the kinds of data, reported 
and interpreted, Unfortunately, program evaluation efforts at the local 
school district level often become "added assignments," that is, the 
follow-up is undertaken amidst the continuing pressures of the ongoing 
instructional program with whatever time is available. Within these 
limitations, however , some school districts have undertaken fairly exten- 
sive surveys of the student population. Three case studies are appended 
here to give some idea of the kinds of evaluation being done. 

Jjnj on 'll •! ph S ch o o 1 District No. 5, Mil w aukie, Oregon . The District 
is comprised of three comprehensive high schools and an Occupational 
Skills Center, which provides vocational education for junior and senior 
students bussed from the. comprehensive high schools. 1 irst— year students 
in the Center attend two hours each day; second-year students attend for 
three hours each day. Part-time work opportunities are available to 
students who have taken some of their course work during the preceding 
summer, thereby accelerating their academic preparation prior to joining 
the work experience program. Students receive pay for their services 
and part-time jobs are available both within the school district and from 
outside employers. 

The District had a total of 836 graduates in 1968 and 887 graduates 
in 1969. Of these, 115 were graduates of the Occupational Skills Center 
in 1968 and 187 were graduates in 1969. Industrial Arts classes (offered 
to freshmen and sophomores) and Business Education and Home Economics 
classes are taught in the home high schools and are not included in Occu- 
pational Skills Center figures. Occupational skills clusters offered at. 
the Center include: Agriculture, Building Construction, Child Services, 

Data Processing and Business Machines, Distributive Education, Electricity 
and Electronics, Graphic Reproduction, Health Services, Industrial 
Mechanics, and Metal Fabrication. 
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Enrol. linen t at the Occupational Skills Center is on the recommendation 
of the sending-school’s principal > upon advice of the counselor and with 
the. concurrence of the student and his parents . The Center’s principal 
has the. option to return students to the home school if they do not seem 
able to adjust to the vocational program as conducted at the Center. 

The instructional staff at the Center include one Director , one 
Placement and Guidance Coordinator, thirteen teachers and eight teacher 
aides. Costs in the comprehensive high schools are $123.37 per student 
hour as compared to $211,37 per student hour in the Center. 

The counseling and career orientation procedures begin in October 
for a senior class, followed by a survey of seniors in April, May, and 
June to determine their post-high school expectations. 



Follow-up is undertaken for all District graduates and an additional 
follow— up is undertaken for Occupational Skills Center graduates. ihe 
initial follow-up is by phone, with each high school counselor calling the 
students assigned to him in the previous year. After three attempts to 
reach the graduate or his parents, other knowledgeable sources are 
contacted. Using this technique, a 95.5% completion of follow-ups was 
accomplished dis triet*~wide in 3.969. The categories of data collected show 

the number of graduates : 

Attending 4-year state schools (in state) 

Attending 4-year state schools (out of state) 

Attending private schools (in or out of state) 

Attending Community College (in state) 

Attending Community/ Junior College, (out of state) 

Entered military service 



Entered state or union apprenticeship training programs 
(does not include OJT provided by the employer) 



Attending other schools (beauty, business, etc.) 

Entered full-time employment 
Entered part-time employment 
Unemployed 

Married (homemaker) ’ 

Other 

Unknown (no response, moved— -no forwarding address, etc.) 
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The Occupational Skills Center students also receive, a mailed 
follow-up survey . It provides data showing the number of students who: 

Entered full-time employment (does it relate, to cluster taken?) 



Entered college (full- or part-time, and does it relate to 
cluster taken?) 

Entered military service * 

Entered state or union apprenticeship training programs (does 
not include OJT provided by the employer) 



A minimum of 20% return of these four-page mailed questionnaires is 
considered acceptable by the Occupational Skills Center personnel. Much 
of the information obtained by this questionnaire is intended to guide, 
the instructional staff as they revise the curriculum in the various 
occupational clusters. Data obtained in these questionnaires are pro- 
cessed by students in the Center’s Data Processing and Business Machines 
occupational cluster, providing an even more direct input to the ongoing 
program activities. 



In spite of a thorough follow-up procedure and the publication of 
reports to state authorities and separately in booklet form to District 
personnel, the District does not have available comparisons between Occu- 
pational Skills Center graduates and any control group of matched or 
similar students. Instead, the comparative emphasis rests on looking at 
the placement of 1969 graduates in relation to the placement a year 
earlier. As pointed out elsewhere in this report, this procedure assumes 
a stable job market in the two years, consistency in draft sign-ups and 
college enrollments, and so on. The same constraints will exist for the 
three- and five-year follow-ups now planned for this program; 

Two other ongoing self-evaluation efforts are undertaken by the 
District. The first is a Release-Time Training Evaluation form which is 
completed by the cooperating employers for those students involved in 
work-experience opportunities. Employers are asked, each spring, to rate 
the student employees on the criteria of (!) punctuality and attendance, 
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(2) initiative, (3) job performance, (4) .skill development , (5) personal 
attitude, and (6) relationship with other employees. They are encouraged 
to make additional comments and suggestions. 

The second form is the Instructor Rating and Recommendation, which 
is completed in the fall and again in the spring by the instructional 
staff. Each student is rated according to (1) development of craftsman- 
ship, (2) technical understanding, (3) demonstrated safety habits, 

(4) cooperation with students and instructor, (5) willingness to accept 
criticism, and (6) personal attitude. Information of this type is intended 
to be used by the Placement and Guidance Coordinator as he attempts to 
place students in appropriate job opportunities. 



Vs* 



Curric u lum Dev elo pment Program 
established to identify and counsel 



vV * 

( CDP). Quincy . Illinois . CDF was . 
the dropout-prone student, to provide 



special classes and practical classroom experiences, to develop a work- 
study program and to involve the parents of dropout -prone students in the 
school program. Socioeconomic status, intelligence, school achievement, 
reading achievement, and. school and social adjustment (as rated, on a des- 
cription sheet of aggressiveness and withdrawnness behavior) were the 
factors used to identify dropout-prone students who were about to enter 
the seventh grade. CDP became a department within the public comprehensive 
school (grades 7-12) and within a few years the program included about 300 
students, 14 full-time teachers, and 5 part-time teachers. It included 
regularly scheduled parent-teacher meetings, home visits by the teachers, 
a parent newsletter, and biweekly in-service teacher training sessions. 
Sheltered work stations were developed within the CDP to provide pre- 
employment experiences and training for students. One of the sheltered 
work stations is called the Service Station Training School, and is a work 
station which is being systematically evaluated. 



The Service Station Training School for CDP boys aged 16 years or 
older is a sheltered work laboratory developed to give the. youths individ- 
ualized p re -employment classroom training and job training to articulate 
and facilitate the transition of the CDP students between the school and th 



world of work. 



The program was intended to provide 



experiences vital to 






retaining students in school until they are ready to seek and maintain 
full-time employment* It was designed to systematically evaluate stu- 
dents T progress 'toward those goals. The students receive both school 
credit and a salary for working part-time at the service station which 
was rented for the purposes of the program. The amount of training 
received during each school year varies with the achievement made by 
each student. Each boy is evaluated monthly by the work experience 
coordinator, the training school instructor, and the work supervisor. 

Each year a number of boys in CDP were eligible for' service station 
training but: only a portion of those boys could be served by the limited 
service station facilities. Of those students who were eligible, a 
number were randomly chosen for service station training and. a number 
were chosen as a control group. Those who did not participate, in the 
station training participated in the regular CDP training. At the end 
of each school year the experimental (or service station) and control 
groups could be compared on these factors: intelligence, reading 

achievement , attitudes, school attendance, grades, and placement in 
employment or further education. At the time of the writing of this 
report a full comparison of the placement in employment or further edu- 
cation of the members of both the experimental and control groups had 
not yet been completed. 



A * * 

The Center f or Vocational Arts (CVA) , Norwalk, Connect icut . CVA is 
a program for school-alienated youths (ages 15-21) which was developed 
to give the youths occupational training and academic instruction, and 
to effect the behavioral changes necessary for obtaining and keeping a 
job. Training is held at the Center and is offered in automotive ser- 
vices, food services, health services, landscaping and horticulture, 
office services, maintenance and repair operations, manufacturing opera- 
tions, and retailing services. The Center emphasizes individualized 
instruction and individualized guidance in both vocational and academic 
areas and is developing, with the help of educational specialists, 
individualized learning activity packages for use in each of the training 
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1uut raaolKul a competency level which marlin a vocational certificate, 
a high school diploma, or both. 

OVA haft participated in evaluation procedures performed by outside 
profess! on el consult on to and local program personnel at various points 
:i n the* school’s history, The following test instruments have at times 
been used to record the- presence of the student attributes listed with 
each instrument : California Study Methods Survey— school adjustment, 

and study habits and skills; California Test of Personality— personal 
and social adjustment} L o r ge~Tho rnd ike Intelligence Tost (Verbal and 
Non-Verb al) — academic aptitude; California Reading Test— reading vocab- 
ulary and reading comprehension; NYU Speaking Test— clarity of speech 
in reading and conversing; Occupational Adjustment and Rating Scale- 
self-concept, social behavior, attitude toward society, school adjustment, 
work attitudes; Fundamental Achievement Series— -use of practical verbal 
and numerical symbols. 

In addition, three studies were made to assess the ways in which 
students perceived their OVA experience: a comparative study of the per- 

ception of the school by the students, 1967-1968; how students at the 
Center saw themselves; and case study reports on the students, 1967- 



To measure the effect of the school on the employability of its 
students, a comparison was made between the job record of some CVA grad- 
uates (June 1966 to April 1969) and the job record of some students 
leaving CVA without completing a program. According to these records, 
88% of the CVA graduates were employed full-time as compared with 73% of 
the nongraduates. Early in the history of the school it was planned 
that there would be a matched control group of students who qualified 
to be in CVA but who would not be admitted. Such a group, when followed 
over a period of years, might have provided a reliable basis against 
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A ccmfoi c*uv cui th f * study of aulcct od u^c r t d,r*ry vocational cuhujnti on 
programs vow Lc-ld at the*. Amen; icon In a kit til* ea for Ron cal' eh , Palo Alto, 
California, on July 10 and 11, 1969. 

Consultants. p tenant at the conference wore as follows. 

Mr, Joseph Re Hanger 
A s s .1, s t an t: S u p a r in t en dent 
San Jose. Unified School District 
1605 Park Avenue 

San Jose, California 95114 . 

Dr, Max U, Enlnger, President: 

Educational Systems Research Institute, Inc. 

. 4527 Ninth top Street 

Pittsburgh, Pennsylvania 15213 



Dr. Jack Mi ch i. e 

San Diego State College 

Sail Diego, California 92115 



Dr. Bytl R. Shoemaker 
Director of Vocational Education 
Department of Education 
612 State Office Building 
Columbus, Ohio 43215 

Dr. Robert M. Worthington 

Assistant Commissioner of Education 

New Jersey State Division of Vocational Education 

P.0. Box 2019 

Trenton, New Jersey 08625 

Apologies were received from the following consultants who were invited 
but. were unable to attend: 

Dr. Melvin Bat'low, UCLA 

Dr. Lowell Burkett, American Vocational Association 

Dr. Rupert Evans, University of Illinois 

Mr, Marvin Feldman, Ford Foundation 

Dr. Edward Morrison, Ohio State University 

Mr. Michael Russo, Division of Vocational and Technical 
Education, U.S. Office of Education 
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ABSTRACT 

The aims were to 'identify, select, analyze, and describe vocational programs 
at the secondary level, that had been successful in increasing the total placement 
rate (employment. , further education) when compared with other coin. ses of instruc- 
tion. Through a literature search, mail and telephone inquiries, personal con- 
tacts, and other means, 445 programs were identified for study with emphasis on 
program evaluation, particularly follow-up of graduates. Some 30 programs were 
site-visited. 

None of the 445 programs could be shown to have met the study’s criteria for 
success; 43% lacked comprehensive follow-up data on graduates; 33% had nearly 
complete data, but no trends could be detected, nor were suitable comparison 
groups of non vocational graduates available. 

Even if comparable, contemporaneous groups had existed, and even if complete 
follow-up of both groups of graduates had been possible, it was "concluded that 
the original criteria needed to be supplemented by cuteria which toon into 
account qualitative placement factors, such as initial job earnings, on tee joo 
competency, job retention, progress within the job, and mobility within a. range 
of jobs related to the training provided by the vocational program. 

Recommendations for program design and evaluation that would permit tns 
assessment of such factors conclude the report. 
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